Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v2.0: Reclaims more old accounts in clean (backport of #4044) #4089

Merged
merged 2 commits into from
Dec 18, 2024
Merged

v2.0: Reclaims more old accounts in clean (backport of #4044) #4089

merged 2 commits into from
Dec 18, 2024

Conversation

mergify[bot]
Copy link

@mergify mergify bot commented Dec 12, 2024 •
edited by brooksprumo
Loading

Problem

In clean_accounts(), if there are old storages and old accounts beyond what is expected, we do not always reclaim them. This is because when scanning the candidates during clean, if the account's latest slot in the slot list is not in the uncleaned_roots list, we don't collect reclaims.

This can happen if there are candidates from old/ancient slots, where their newest entry is in a slot that is not being visited by clean. We would find multiple entries for this candidate in the index, yet determine not to collect reclaims.

Summary of Changes

Collect reclaims on candidates that have slot lists greater than one entry, and if they are all cleanable (i.e. rooted).

Justification to Backport

Without this change, nodes running with --disable-accounts-disk-index will collect excessive storages, because startup index generation does not correctly mark duplicate pubkeys when the disk index is disabled. But now clean does go and inspect those old storages, yet it won't reclaim the old versions of the accounts because the newest version of the account is not in the uncleaned_roots list. And because we don't reclaim these old versions, we cannot purge old storages that are all dead.

(Historically the uncleaned_roots were all the most recent slots, so it made sense to check for the newest versions there. Now, we clean really old slots too, which means the newest version of accounts in these old slots may be elsewhere (i.e. not just in uncleaned_roots))

This is an automatic backport of pull request #4044 done by [Mergify](https://mergify.com).

@brooksprumo @mergify
Reclaims more old accounts in clean (#4044)
1102014 
(cherry picked from commit 3d43824)

# Conflicts:
#	accounts-db/src/accounts_db.rs
#	accounts-db/src/accounts_db/tests.rs
@mergify mergify bot requested a review from a team as a code owner December 12, 2024 19:56
@mergify mergify bot added the conflicts label Dec 12, 2024
@mergify Mergify
Copy link
Author

mergify bot commented Dec 12, 2024

Cherry-pick of 3d43824 has failed:

On branch mergify/bp/v2.0/pr-4044
Your branch is up to date with 'origin/v2.0'.

You are currently cherry-picking commit 3d43824154.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   accounts-db/src/accounts_index.rs

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   accounts-db/src/accounts_db.rs
	deleted by us:   accounts-db/src/accounts_db/tests.rs

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

HaoranYi
HaoranYi reviewed Dec 13, 2024
View reviewed changes
Copy link

@HaoranYi HaoranYi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.
One question, are those test code necessary for backporting?

@brooksprumo
Copy link

One question, are those test code necessary for backporting?

The tests aren't necessary for the logic changes, but seems like a good idea to also keep the tests for backports in general, right?

@brooksprumo brooksprumo requested a review from HaoranYi December 16, 2024 19:14
HaoranYi
HaoranYi approved these changes Dec 16, 2024
View reviewed changes
Copy link

@HaoranYi HaoranYi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core logic change looks good to me.

The test is only partially test what we intended to test. There is a follow up PR which improve these tests.

#4147

Since we are backporting the new tests here, maybe we want to update the tests here to use 4147 in this back port PR?

@brooksprumo
Copy link

Since we are backporting the new tests here, maybe we want to update the tests here to use 4147 in this back port PR?

Yeah, that's a good point. I'm not sure if it's best to pull the test fixes into this PR, or to not and then backport the fixes on their own.

@HaoranYi
Copy link

HaoranYi commented Dec 16, 2024 •
edited
Loading

yeah.

I think the goal for include these tests in the backport is to make sure we are testing the clean for old slots even if the new slot is not added. For the test to exercise this, we really need 4147.

However, creating another backport for just the sake of these tests doesn't seem to worth it,
My suggestion is just to cherry-pick 4147 on top of this backport...

@brooksprumo
Copy link

I'm going to merge this backport as-is, and then backport the tests on their own. This keeps the PRs on master/v2.1/v2.0 the same, which is useful for any debug/ops-related reverts/bisects/etc.

@brooksprumo brooksprumo merged commit efd7e87 into v2.0 Dec 18, 2024
38 checks passed
@brooksprumo brooksprumo deleted the mergify/bp/v2.0/pr-4044 branch December 18, 2024 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@t-nelson t-nelson t-nelson approved these changes

@HaoranYi HaoranYi HaoranYi approved these changes

@jeffwashington jeffwashington Awaiting requested review from jeffwashington

Assignees

@brooksprumo brooksprumo

Labels
conflicts
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@brooksprumo @HaoranYi @t-nelson